Sylvain Beucler: Practical basics of reproducible builds 2
Let's review what we learned so far:
It turns out that PE carries a build date timestamp. The spec says that bound DLLs timestamps are refered to in the "Delay-Load Directory Table". Maybe that's also the date Windows displays when a system-wide DLL is about to be replaced, too.
Build timestamps looks unused in .exe files though. Anyway, Stephen Kitt pointed out (thanks!) that Debian's MinGW linker binutils-mingw-w64 has an upstream-pending patch that sets the timestamp to SOURCE_DATE_EPOCH if set. Alternatively, one can pass
If we don't care about debug symbols, unlike with ELF, stripped PE binaries look stable too!
Now that we have the main executable covered, what about the dependencies?
Let's see how well MXE compiles SDL2:
Neat - what about another build directory?
Well that was expected.
But what about the filesystem order?
With such an automated build, could potential variations in the order of files go undetected?
Would the output be different on another filesystem format (ext4 vs. btrfs...)? It was a good opportunity to test the disorderfs fuse-based tool.
And while I'm at it, check if reprotest is easy enough to use (the manpage is scary).
Let's redo our basic tests with it - basic usage is actually very simple:
Now let's apply to the MXE build.
We keep the same build path, and also avoid using
That checks!
What about a full MXE environment?
Most libraries look reproducible enough.
We can compile with
There are various issues (timestamps, permissions) but first let's check the executables themselves:
Gah...
AFAIU there is something random in the linking phase, and sometimes the timestamp is removed, sometimes it's not.
Not very easy to track but I believe I reproduced it with the "hello" example:
It looks like there is a random behavior in binutils 2.25, coupled with SDL2's wrapping of my
Yay!
I could not reproduce the build timestamp issue in the stripped binaries, though it was still varying in the unstripped
I mentioned there was other changes noticed by diffoscope.
Could be ignored, but we want to generate an identical binary package/archive too, right?
That's where archive meta-data matters.
The doc suggests clamping all files to SOURCE_DATE_EPOCH - i.e. all generated files will have their date set at that timestamp:
I attempted to mitigate the issue by playing with
However even
Maybe it's better to set and record the umask...
So, aside from minor issues such as BuildIDs and build timestamps, the toolchain is pretty stable as of now.
The issue is more about fixing and recording the build environment.
Which is probably the next challenge
- compiler version need to be identical and recorded
- build options and their order needs to be identical and recorder
- build path needs to be identical and recorded
(otherwise debug symbols - and BuildIDs - change) - diffoscope helps checking for differences in build output
It turns out that PE carries a build date timestamp. The spec says that bound DLLs timestamps are refered to in the "Delay-Load Directory Table". Maybe that's also the date Windows displays when a system-wide DLL is about to be replaced, too.
Build timestamps looks unused in .exe files though. Anyway, Stephen Kitt pointed out (thanks!) that Debian's MinGW linker binutils-mingw-w64 has an upstream-pending patch that sets the timestamp to SOURCE_DATE_EPOCH if set. Alternatively, one can pass
-Wl,--no-insert-timestamp to set it to 0 (though see caveats below):
$ i686-w64-mingw32.static-gcc -Wl,--no-insert-timestamp hello.c -o hello.exe
$ md5sum hello.exe
298f98d74e6e913628a8b74514eddcb2 hello.exe
$ /opt/mxe/usr/bin/i686-w64-mingw32.static-gcc -Wl,--no-insert-timestamp hello.c -o hello.exe
$ md5sum hello.exe
298f98d74e6e913628a8b74514eddcb2 hello.exe
$ cd repro/
$ i686-w64-mingw32.static-gcc hello.c -o hello.exe && i686-w64-mingw32.static-strip hello.exe
$ md5sum hello.exe
6e07736bf8a59e5397c16e799699168d hello.exe
$ i686-w64-mingw32.static-gcc hello.c -o hello.exe && i686-w64-mingw32.static-strip hello.exe
$ md5sum hello.exe
6e07736bf8a59e5397c16e799699168d hello.exe
$ cd ..
$ cp -a repro repro2/
$ cd repro2/
$ i686-w64-mingw32.static-gcc hello.c -o hello.exe && i686-w64-mingw32.static-strip hello.exe
$ md5sum hello.exe
6e07736bf8a59e5397c16e799699168d hello.exe
Let's see how well MXE compiles SDL2:
$ cd /opt/mxe/
$ cp -a ./usr/i686-w64-mingw32.static/lib/libSDL2.a /tmp
$ rm -rf * && git checkout .
$ make sdl2
$ md5sum ./usr/i686-w64-mingw32.static/lib/libSDL2.a /tmp/libSDL2.a
68909ab13181b1283bd1970a56d41482 ./usr/i686-w64-mingw32.static/lib/libSDL2.a
68909ab13181b1283bd1970a56d41482 /tmp/libSDL2.a
$ cd /usr/srx/mxe
$ make sdl2
$ md5sum usr/i686-w64-mingw32.static/lib/libSDL2.a /tmp/libSDL2.a
c6c368323927e2ae7adab7ee2a7223e9 usr/i686-w64-mingw32.static/lib/libSDL2.a
68909ab13181b1283bd1970a56d41482 /tmp/libSDL2.a
$ ls -l ./usr/i686-w64-mingw32.static/lib/libSDL2.a /tmp/libSDL2.a
-rw-r--r-- 1 me me 5861536 mars 23 21:04 /tmp/libSDL2.a
-rw-r--r-- 1 me me 5862488 mars 25 19:46 ./usr/i686-w64-mingw32.static/lib/libSDL2.a
But what about the filesystem order?
With such an automated build, could potential variations in the order of files go undetected?
Would the output be different on another filesystem format (ext4 vs. btrfs...)? It was a good opportunity to test the disorderfs fuse-based tool.
And while I'm at it, check if reprotest is easy enough to use (the manpage is scary).
Let's redo our basic tests with it - basic usage is actually very simple:
$ apt-get install reprotest disorderfs faketime
$ reprotest 'make hello' 'hello'
...
will vary: environment
will vary: fileordering
will vary: home
will vary: kernel
will vary: locales
will vary: exec_path
will vary: time
will vary: timezone
will vary: umask
...
--- /tmp/tmpk5uipdle/control_artifact/
+++ /tmp/tmpk5uipdle/experiment_artifact/
--- /tmp/tmpk5uipdle/control_artifact/hello
+++ /tmp/tmpk5uipdle/experiment_artifact/hello
stat
@@ -1,8 +1,8 @@
Size: 8632 Blocks: 24 IO Block: 4096 regular file
Links: 1
-Access: (0755/-rwxr-xr-x) Uid: ( 1000/ me) Gid: ( 1000/ me)
+Access: (0775/-rwxrwxr-x) Uid: ( 1000/ me) Gid: ( 1000/ me)
Modify: 1970-01-01 00:00:00.000000000 +0000
Birth: -
# => OK except for permissions
$ reprotest 'make hello && chmod 755 hello' 'hello'
=======================
Reproduction successful
=======================
No differences in hello
c8f63b73265e69ab3b9d44dcee0ef1d2815cdf71df3c59635a2770e21cf462ec hello
$ reprotest 'make hello CFLAGS="-g -O2"' 'hello'
# => lots of differences, as expected
We keep the same build path, and also avoid using
linux32 (because MXE would then recompile all the host compiler tools for 32-bit):
$ reprotest --dont-vary build_path,kernel 'touch src/sdl2.mk && make sdl2 && cp -a usr/i686-w64-mingw32.static/lib/libSDL2.a .' 'libSDL2.a'
=======================
Reproduction successful
=======================
No differences in libSDL2.a
d9a39785fbeee5a3ac278be489ac7bf3b99b5f1f7f3e27ebf3f8c60fe25086b5 libSDL2.a
What about a full MXE environment?
$ reprotest --dont-vary build_path,kernel 'make clean && make sdl2 sdl2_gfx sdl2_image sdl2_mixer sdl2_ttf libzip gettext nsis' 'usr'
# => changes in installation dates
# => timestamps in .exe files (dbus, ...)
# => libicu doesn't look reproducible (derb.exe, genbrk.exe, genccode.exe...)
# => apparently ar timestamp variations in libaclui
ar differences may go away at FreeDink link time since I'm aiming at a static build.
Let's try!
First let's see how FreeDink behaves with stable dependencies.We can compile with
-Wl,--no-insert-timestamp and strip the binaries in a first step.There are various issues (timestamps, permissions) but first let's check the executables themselves:
$ cd freedink/
$ reprotest --dont-vary build_path 'mkdir cross-woe-32/ && cd cross-woe-32/ && export PATH=/opt/mxe/usr/bin:$PATH && LDFLAGS='-Wl,--no-insert-timestamp' ../configure --host=i686-w64-mingw32.static --enable-static && make -j$(nproc) && make install-strip DESTDIR=$(pwd)/destdir' 'cross-woe-32/destdir/usr/local/bin'
# => executables are identical!
# Same again, just to make sure
$ reprotest --dont-vary build_path 'mkdir cross-woe-32/ && cd cross-woe-32/ && export PATH=/opt/mxe/usr/bin:$PATH && LDFLAGS='-Wl,--no-insert-timestamp' ../configure --host=i686-w64-mingw32.static --enable-static && make -j$(nproc) && make install-strip DESTDIR=$(pwd)/destdir' 'cross-woe-32/destdir/usr/local/bin'
--- /tmp/tmp2yw0sn4_/control_artifact/bin/freedink.exe
+++ /tmp/tmp2yw0sn4_/experiment_artifact/bin/freedink.exe
@@ -2,20 +2,20 @@
00000010: b800 0000 0000 0000 4000 0000 0000 0000 ........@.......
00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000030: 0000 0000 0000 0000 0000 0000 8000 0000 ................
00000040: 0e1f ba0e 00b4 09cd 21b8 014c cd21 5468 ........!..L.!Th
00000050: 6973 2070 726f 6772 616d 2063 616e 6e6f is program canno
00000060: 7420 6265 2072 756e 2069 6e20 444f 5320 t be run in DOS
00000070: 6d6f 6465 2e0d 0d0a 2400 0000 0000 0000 mode....$.......
-00000080: 5045 0000 4c01 0a00 e534 0735 0000 0000 PE..L....4.5....
+00000080: 5045 0000 4c01 0a00 0000 0000 0000 0000 PE..L...........
00000090: 0000 0000 e000 0e03 0b01 0219 00f2 3400 ..............4.
000000a0: 0022 4e00 0050 3b00 c014 0000 0010 0000 ."N..P;.........
000000b0: 0010 3500 0000 4000 0010 0000 0002 0000 ..5...@.........
000000c0: 0400 0000 0100 0000 0400 0000 0000 0000 ................
-000000d0: 00e0 8900 0004 0000 7662 4e00 0200 0000 ........vbN.....
+000000d0: 00e0 8900 0004 0000 89f8 4e00 0200 0000 ..........N.....
000000e0: 0000 2000 0010 0000 0000 1000 0010 0000 .. .............
000000f0: 0000 0000 1000 0000 00a0 8700 b552 0000 .............R..
00000100: 0000 8800 d02d 0000 0050 8800 5006 0000 .....-...P..P...
00000110: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000120: 0060 8800 4477 0100 0000 0000 0000 0000 . ..Dw..........
00000130: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000140: 0440 8800 1800 0000 0000 0000 0000 0000 .@..............
stat
@@ -1,8 +1,8 @@
Size: 5121536 Blocks: 10008 IO Block: 4096 regular file
Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 1000/ me) Gid: ( 1000/ me)
-Modify: 2017-03-26 01:26:35.233841833 +0000
+Modify: 2017-03-26 01:27:01.829592505 +0000
Birth: -
AFAIU there is something random in the linking phase, and sometimes the timestamp is removed, sometimes it's not.
Not very easy to track but I believe I reproduced it with the "hello" example:
# With MXE:
$ reprotest 'i686-w64-mingw32.static-gcc hello.c -I /opt/mxe/usr/i686-w64-mingw32.static/include -I/opt/mxe/usr/i686-w64-mingw32.static/include/SDL2 -L/opt/mxe/usr/i686-w64-mingw32.static/lib -lmingw32 -Dmain=SDL_main -lSDL2main -lSDL2 -lSDL2main -Wl,--no-insert-timestamp -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lversion -o hello && chmod 700 hello' 'hello'
# => different
# => maybe because it imports the build timestamp from -lSDL2main
# With Debian's MinGW (but without SOURCE_DATE_EPOCH):
$ reprotest 'i686-w64-mingw32-gcc hello.c -I /opt/mxe/usr/i686-w64-mingw32.static/include -I/opt/mxe/usr/i686-w64-mingw32.static/include/SDL2 -L/opt/mxe/usr/i686-w64-mingw32.static/lib -lmingw32 -Dmain=SDL_main -lSDL2main -lSDL2 -lSDL2main -Wl,--no-insert-timestamp -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lversion -o hello && chmod 700 hello' 'hello'
=======================
Reproduction successful
=======================
No differences in hello
0b2d99dc51e2ad68ad040d90405ed953a006c6e58599beb304f0c2164c7b83a2 hello
# Let's remove -Dmain=SDL_main and let our main() have precedence over the one in -lSDL2main:
$ reprotest 'i686-w64-mingw32.static-gcc hello.c -I /opt/mxe/usr/i686-w64-mingw32.static/include -I/opt/mxe/usr/i686-w64-mingw32.static/include/SDL2 -L/opt/mxe/usr/i686-w64-mingw32.static/lib -lmingw32 -lSDL2main -lSDL2 -lSDL2main -Wl,--no-insert-timestamp -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lversion -o hello && chmod 700 hello' 'hello'
=======================
Reproduction successful
=======================
No differences in hello
6c05f75eec1904d58be222cc83055d078b4c3be8b7f185c7d3a08b9a83a2ef8d hello
$ LANG=C i686-w64-mingw32.static-ld --version # MXE
GNU ld (GNU Binutils) 2.25.1
Copyright (C) 2014 Free Software Foundation, Inc.
$ LANG=C i686-w64-mingw32-ld --version # Debian
GNU ld (GNU Binutils) 2.27.90.20161231
Copyright (C) 2016 Free Software Foundation, Inc.
main().
So FreeDink is nearly reproducible, except for this build timestamp issue that pops up in all kind of situations.
In the worse case I can zero it out, or patch MXE's binutils until they upgrade.
More importantly, what if I recompile FreeDink and the dependencies twice?
$ (cd /opt/mxe/ && make clean && make sdl2 sdl2_gfx sdl2_image sdl2_mixer sdl2_ttf glm libzip gettext nsis)
$ (mkdir cross-woe-32/ && cd cross-woe-32/ \
&& export PATH=/opt/mxe/usr/bin:$PATH \
&& LDFLAGS="-Wl,--no-insert-timestamp" ../configure --host=i686-w64-mingw32.static --enable-static \
&& make V=1 -j$(nproc) \
&& make install-strip DESTDIR=$(pwd)/destdir)
$ mv cross-woe-32/ cross-woe-32-1/
# Same again...
$ mv cross-woe-32/ cross-woe-32-2/
$ diff -ru cross-woe-32-1/destdir/ cross-woe-32-2/destdir/
[nothing]
I could not reproduce the build timestamp issue in the stripped binaries, though it was still varying in the unstripped
src/freedinkedit.exe.
I mentioned there was other changes noticed by diffoscope.
- Changes in file timestamps.
Could be ignored, but we want to generate an identical binary package/archive too, right?
That's where archive meta-data matters.
make INSTALL="$(which install) install -p" could help for static files, but not generated ones.The doc suggests clamping all files to SOURCE_DATE_EPOCH - i.e. all generated files will have their date set at that timestamp:
$ export SOURCE_DATE_EPOCH=$(date +%s) \
&& reprotest --dont-vary build_path \
'make ... && find destdir/ -newermt "@$ SOURCE_DATE_EPOCH " -print0 xargs -0r touch --no-dereference --date="@$ SOURCE_DATE_EPOCH "' 'cross-woe-32/destdir/'
- Changes in directory permissions
I attempted to mitigate the issue by playing with
make install MKDIR_P="mkdir -p -m 755" (1).However even
mkdir -p -m ... does not set permissions for intermediate directories.Maybe it's better to set and record the umask...
So, aside from minor issues such as BuildIDs and build timestamps, the toolchain is pretty stable as of now.
The issue is more about fixing and recording the build environment.
Which is probably the next challenge
Here is my monthly update covering what I have been doing in the free software world (
An updated 
I m investigating the creation of VM images for different virtualisation solutions.
Among the target platforms is a destop as a service platform based on an OpenStack public cloud.
We ve been working with 
Better late than never, we'd like to welcome our three Outreachy interns
for this round, lasting from the 6th of December 2016 to the 6th of March 2017.
2 more weeks of my awesome Outreachy journey have passed, so it is time to make an update on my progress.
I continued my work on improving diffoscope by fixing bugs and completing wishlist items. These include:
So, it turns out when you 
I m at an airport, about to board the first of three flights across the world, from timezone +8 to timezone -8. I ll be in transit 27 hours to get to Seattle, Washington state. I m leaving my wife and two young children behind.
My work has given me a days worth of leave under the Corporate Social Responsibility program, and I m taking three days annual leave, to do this. 27 hours each way in transit, for 3 days on the ground.
Why?
Backstory
I started playing in technology as a kid in the 1980s; my first PC was a clone (as they were called) 286 running MS-DOS. It was clunky, and the most I could do to extend it was to write batch scripts. As a child I had no funds for commercial compilers, no network connections (this was pre Internet in Australia), no access to documentation, and no idea where to start programming properly.
It was a closed world.
I hit university in the summer of 1994 to study Computer Science and French. I d heard of Linux, and soon found myself installing the Linux distributions of the day. The Freedom of the licensing, the encouragement to use, modify, share, was in stark contrast to the world of consumer PCs of the late 1980 s.
It was there at the UCC at UWA I discovered Debian. Some of the kind network/system admins at the University maintained a Debian mirror on the campus LAN, updated regularly and always online. It was fast, and more importantly, free for me to access. Back in the 1990s, bandwidth in Australia was incredibly expensive. The vast distances of the country mean that bandwidth was scarce. Telcos were in races to put fiber between Perth and the Eastern States, and without that in place, IP connectivity was constrained, and thus costly.
Over many long days and nights I huddled down, learning window managers, protocols, programming and scripting languages. I became a system/network administrator, web developer, dev ops engineer, etc. My official degree workload, algorithmic complexity, protocol stacks, were interesting, but fiddling with Linux based implementations was practical.
Volunteer
After years of consuming the output of Debian and running many services with it I decided to put my hand up and volunteer as a Debian Developer: it was time to give back. I had benefited from Debian, and I saw others benefit from it as well.
As the 2000 s started, I had my PGP key in the Debian key ring. I had adopted a package and was maintaining it load balancing Apache web servers. The web was yet to expand to the traffic levels you see today; most web sites were served from one physical web server. Site Reliability Engineering was a term not yet dreamed of.
What became more apparent was the applicability of Linux, Open Source, and in my line-of-sight Debian to a wider community beyond myself and my university peers. Debain was being used to revive recycled computers that were being donated to charities; in some cases, unable to transfer commercial software licenses with the hardware that was no longer required by organisations that had upgraded. It appeared that Debian was being used as a baseline above which society in general had access to fundamental capability of computing and network services.
The removal of subscriptions, registrations, and the encouragement of distribution meant this occurred at rates that could never be tracked, and more importantly, the consensus was that it should not be automatically tracked. The privacy of the user is paramount more important than some statistics for the Developer to ponder.
When the Bosnia-Herzegovina war ended in 1995, I recall an email from academics there, having found some connectivity, writing to ask if they would be able to use Debian as part of their re-deployment of services for the Tertiary institutions in the region. This was an unnecessary request as Debian GNU/Linux is freely available, but it was a reminder that, for the country to have tried to procure commercial solutions at that time would have been difficult. Instead, those that could do the task just got on with it.
There s been many similar project where the grass-roots organisations non profits, NGOs, and even just loose collectives of individuals have turned to Linux, Open Source, and sometimes Debian to solve their problems. Many fine projects have been established to make technology accessible to all, regardless of race, gender, nationality, class, or any other label society has used to divide humans. Big hat tip to 